Comparing Methodologies in a Series of Speech Outcome Studies : Challenges and Lessons Learned.

BACKGROUND: Development of the speech audit tool Cleft Audit Protocol for Speech Augmented (CAPS-A) facilitated intercenter comparison of speech outcomes following cleft palate repair. The CAPS-A protocol recommends consensus listening by 3 speech and language therapists, 2 of whom must be CAPS-A trained. Allowing 15 minutes per sample, 15 to 20 samples can be assessed each day. Centers typically have resources to audit 15 to 75 samples per year but not to report speech outcomes of larger data sets for research. This 3-phased outcome study examines how the implementation of the CAPS-A protocol might be modified without compromising reliability. METHODOLOGY: In phase 1, 2 external listeners independently rated 42 speech samples; in phase 2, 2 external listeners consensus listened 25% of 140 samples before 1 listener independently rated the remainder; phase 3 compared 124 Great Ormond Street Speech Assessment (GOS.SP.ASS'98) records from live assessments with CAPS-A-rated video samples. RESULTS: Hypernasality, nasal airflow, and passive cleft speech characteristics were rated to identify signs of velopharyngeal dysfunction across all phases. Phase 1 demonstrated suboptimal correlation, intraclass correlation coefficient (ICC) ranging between 0.39 and 0.72. However, the "modified" CAPS-A consensus listening process in phase 2 achieved a mean ICC of 0.91. Phase 3 revealed only moderate agreement between GOS.SP.ASS'98 and CAPS-A. CONCLUSION: A large data set of speech samples was successfully managed by establishing good interrater reliability on 25% of the data, which calibrated the listeners and validated a decision for only 1 of 2 listeners to rate the remaining speech samples. The recommended implementation of the CAPS-A protocol can therefore be modified for more efficient speech outcome reporting.

The Americleft Speech Project: A Training and Reliability Study.

OBJECTIVE: To describe the results of two reliability studies and to assess the effect of training on interrater reliability scores. DESIGN: The first study (1) examined interrater and intrarater reliability scores (weighted and unweighted kappas) and (2) compared interrater reliability scores before and after training on the use of the Cleft Audit Protocol for Speech-Augmented (CAPS-A) with British English-speaking children. The second study examined interrater and intrarater reliability on a modified version of the CAPS-A (CAPS-A Americleft Modification) with American and Canadian English-speaking children. Finally, comparisons were made between the interrater and intrarater reliability scores obtained for Study 1 and Study 2. PARTICIPANTS: The participants were speech-language pathologists from the Americleft Speech Project. RESULTS: In Study 1, interrater reliability scores improved for 6 of the 13 parameters following training on the CAPS-A protocol. Comparison of the reliability results for the two studies indicated lower scores for Study 2 compared with Study 1. However, this appeared to be an artifact of the kappa statistic that occurred due to insufficient variability in the reliability samples for Study 2. When percent agreement scores were also calculated, the ratings appeared similar across Study 1 and Study 2. CONCLUSION: The findings of this study suggested that improvements in interrater reliability could be obtained following a program of systematic training. However, improvements were not uniform across all parameters. Acceptable levels of reliability were achieved for those parameters most important for evaluation of velopharyngeal function.

The Buccinator Mucomuscular Flap: An In-Depth Analysis and Evaluation of Its Role in the Management of Velopharyngeal Dysfunction.

BACKGROUND: A significant proportion of patients with cleft palate experience persisting velopharyngeal dysfunction (VPD) after primary surgery. Pharyngoplasty is the most common procedure to correct inadequate velopharyngeal closure. Although it is often effective, it poses a risk for postoperative airway obstruction. The mucomuscular buccinator flap is a more recent alternative: In the largest case series to date, we outline its use and evaluate its effectiveness in managing persistent VPD. METHODS: Over 9 years, 103 buccinator flap procedures were performed by the Cambridge group to improve velar function. Clinical records were retrospectively assessed: 60 patients were analyzed by two expert speech and language therapists external to the group using the Cleft Audit Protocol for Speech-Augmented. In a subset of patients, the buccinator flap was interpolated between the limbs of a large mucosal Z-plasty. Consensus listening was undertaken, and interrater reliability was calculated for 24.17% of the cohort. The remaining samples were assessed by a single listener following calibrated consensus listening. RESULTS: There was a significant reduction in VPD (P < .001). Preoperatively, 68.5% of patients demonstrated marked VPD, falling to 24.1% postoperatively. The buccinator procedure demonstrated significant improvement for three of four individual speech parameters: hypernasality, passive cleft speech characteristics, and audible nasal emission. Overall, 13.5% of patients required further corrective speech surgery. CONCLUSIONS: The buccinator mucomuscular flap reliably and effectively improves velar function in the management of VPD and has low complication rates. We therefore recommend the use of the buccinator flap in primary surgical management of persisting VPD.

The Importance of Integration of Stakeholder Views in Core Outcome Set Development: Otitis Media with Effusion in Children with Cleft Palate.

BACKGROUND: Approximately 75% of children with cleft palate (CP) have Otitis Media with Effusion (OME) histories. Evidence for the effective management of OME in these children is lacking. The inconsistency in outcome measurement in previous studies has led to a call for the development of a Core Outcome Set (COS). Despite the increase in the number of published COS, involvement of patients in the COS development process, and methods to integrate the views of patients and health professionals, to date have been limited. METHODS AND FINDINGS: A list of outcomes measured in previous research was identified through reviewing the literature. Opinion on the importance of each of these outcomes was then sought from key stakeholders: Ear, Nose and Throat (ENT) surgeons, audiologists, cleft surgeons, speech and language therapists, specialist cleft nurses, psychologists, parents and children. The opinion of health professionals was sought in a three round Delphi survey where participants were asked to score each outcome using a bespoke online system. Parents and children were also asked to score outcomes in a survey and provided an in-depth insight into having OME through semi-structured interviews. The results of the Delphi survey, interviews and parent/patient survey were brought together in a final consensus meeting with representation from all stakeholders. A final set of eleven outcomes reached the definition of "consensus in" to form the recommended COS: hearing; chronic otitis media (COM); OME; receptive language skills; speech development; psycho social development; acute otitis media (AOM); cholesteatoma; side effects of treatment; listening skills; otalgia. CONCLUSIONS: We have produced a recommendation about the outcomes that should be measured, as a minimum, in studies of the management of OME in children with CP. The development process included input from key stakeholders and used novel methodology to integrate the opinion of healthcare professionals, parents and children.

A cross-sectional cohort study of speech in five-year-olds with cleft palate ± lip to support development of national audit standards: benchmarking speech standards in the United Kingdom.

OBJECTIVE: To develop national standards for speech outcomes and processes of care for children with cleft palate ± lip and to test the standards using national data. DESIGN, SETTING, AND PARTICIPANTS: In this large, multicenter, prospective cohort study, speech recordings of 1110 five-year-olds with cleft palate involvement (born 2001 to 2003) were collected by 12 cleft centers in Great Britain and Ireland. Recordings were analyzed by consensus by specialist speech and language therapists using the Cleft Audit Protocol for Speech-Augmented. Results were benchmarked against evidence-based process and speech outcome standards and statistical analysis undertaken. RESULTS: From the 1110 children audited, 48% (530) had speech within the normal range. This was not significantly different from the agreed standard of 50% (P = .20, CI = 45-50%). Sixty-six percent (734) had speech with no evidence of structurally related speech problems or history of speech-related secondary surgery. This was significantly below the standard of 70% (P = .007, CI = 62-69%). Sixty percent (666) had no serious cleft-related articulation errors. This was significantly better than the agreed standard of 50% (P < .001, CI = 67-73%). More than 80% of 2-year-olds received a specialist speech and language assessment against a benchmark of 100%. CONCLUSIONS: Developing standards has facilitated more meaningful reporting of speech outcomes and treatment processes. Evidence-based standards were defined and extensively tested, enabling centers to compare their performance with national trends. One 5-year outcome standard was achievable; the other two standards will require modification through the mandatory annual national audit program.

The cleft audit protocol for speech-augmented: A validated and reliable measure for auditing cleft speech.

OBJECTIVES: To develop an assessment tool for use in intercenter audit studies of cleft speech and to test its acceptability, validity, and reliability. The tool is to be used systematically to record and report speech outcomes, providing an indication of treatment needs and continuing burden of care. SETTING: Regional Cleft Center, U.K. METHODS: The Cleft Audit Protocol for Speech-Augmented (CAPS-A) was developed by three cleft speech experts who identified the key features required from existing assessment measures. Criterion validity was assessed by comparing the Cleft Audit Protocol for Speech-Augmented outcomes reported for 20 cases with clinical assessment results and other investigations. Intra- and interrater reliability were tested following the training of specialist speech and language therapists who used the Cleft Audit Protocol for Speech-Augmented on two occasions to assess 10 cases. The raters evaluated acceptability and ease of using a questionnaire. RESULTS: The mean percentage agreement for criterion validity in each section was 87% (range 70% to 100%). Both intra- and interexaminer reliability were rated as good/very good (Kappa 0.61 to 1.00) for seven sections and moderate (Kappa 0.41 to 0.60) for three sections. Raters reported that the Cleft Audit Protocol for Speech-Augmented was acceptable and easy to use with appropriate training. CONCLUSION: An acceptable, valid, and reliable cleft speech audit tool has been developed based on a small sample. The Cleft Audit Protocol for Speech-Augmented is recommended for use in intercenter audit studies in the U.K. and Ireland and could be used in other English-speaking countries. In addition, it has wider applicability for use in reporting speech outcomes of surgical procedures.